{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "pip -q install --upgrade pip\n",
    "pip -q install sagemaker awscli boto3 --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import HTML\n",
    "HTML(\"<script>Jupyter.notebook.kernel.restart()</script>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Direct Marketing with Amazon SageMaker AutoPilot\n",
    "_**Supervised Learning with Gradient Boosted Trees: A Binary Prediction Problem With Unbalanced Classes**_\n",
    "\n",
    "Last update: December 2nd, 2019"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import smdebug_rulesconfig as rule_configs\n",
    "import boto3\n",
    "import os, sys\n",
    "\n",
    "print (sagemaker.__version__)\n",
    "\n",
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()                     \n",
    "prefix = 'sagemaker-autopilot/DEMO-automl-dm'\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "# Role when working on a notebook instance\n",
    "role = sagemaker.get_execution_role()\n",
    "# Role when working locally\n",
    "#role = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm = boto3.Session().client(service_name='sagemaker',region_name=region)\n",
    "sm_rt = boto3.Session().client('runtime.sagemaker', region_name=region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset - Bank Marketing\n",
    "\n",
    "https://archive.ics.uci.edu/ml/datasets/Bank+Marketing\n",
    "\n",
    "### Input variables:\n",
    "#### bank client data:\n",
    "```\n",
    "1 - age (numeric)\n",
    "2 - job : type of job (categorical: 'admin.','blue-collar','entrepreneur','housemaid','management','retired','self-employed','services','student','technician','unemployed','unknown')\n",
    "3 - marital : marital status (categorical: 'divorced','married','single','unknown'; note: 'divorced' means divorced or widowed)\n",
    "4 - education (categorical: 'basic.4y','basic.6y','basic.9y','high.school','illiterate','professional.course','university.degree','unknown')\n",
    "5 - default: has credit in default? (categorical: 'no','yes','unknown')\n",
    "6 - housing: has housing loan? (categorical: 'no','yes','unknown')\n",
    "7 - loan: has personal loan? (categorical: 'no','yes','unknown')\n",
    "```\n",
    "\n",
    "#### related with the last contact of the current campaign:\n",
    "```\n",
    "8 - contact: contact communication type (categorical: 'cellular','telephone')\n",
    "9 - month: last contact month of year (categorical: 'jan', 'feb', 'mar', ..., 'nov', 'dec')\n",
    "10 - day_of_week: last contact day of the week (categorical: 'mon','tue','wed','thu','fri')\n",
    "11 - duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y='no'). Yet, the duration is not known before a call is performed. Also, after the end of the call y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model.\n",
    "```\n",
    "\n",
    "#### other attributes:\n",
    "```\n",
    "12 - campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)\n",
    "13 - pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)\n",
    "14 - previous: number of contacts performed before this campaign and for this client (numeric)\n",
    "15 - poutcome: outcome of the previous marketing campaign (categorical: 'failure','nonexistent','success')\n",
    "```\n",
    "\n",
    "#### social and economic context attributes\n",
    "```\n",
    "16 - emp.var.rate: employment variation rate - quarterly indicator (numeric)\n",
    "17 - cons.price.idx: consumer price index - monthly indicator (numeric)\n",
    "18 - cons.conf.idx: consumer confidence index - monthly indicator (numeric)\n",
    "19 - euribor3m: euribor 3 month rate - daily indicator (numeric)\n",
    "20 - nr.employed: number of employees - quarterly indicator (numeric)\n",
    "```\n",
    "\n",
    "#### output variable (desired target):\n",
    "```\n",
    "21 - y - has the client subscribed a term deposit? (binary: 'yes','no')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -N https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank-additional.zip\n",
    "!unzip -o bank-additional.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read the CSV file into a Pandas data frame and take a look at the first few lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html\n",
    "data = pd.read_csv('./bank-additional/bank-additional-full.csv', sep=';')\n",
    "pd.set_option('display.max_columns', 500)     # Make sure we can see all of the columns\n",
    "pd.set_option('display.max_rows', 50)         # Keep the output on one page\n",
    "data[:10] # Show the first 10 lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.shape # (number of lines, number of columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting the dataset\n",
    "\n",
    "We split the dataset into training (95%) and test (5%) datasets. We will use the training dataset for AutoML, where it will be automatically split again for training and validation.\n",
    " \n",
    "Once the model has been deployed, we'll use the test dataset to evaluate its performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the seed to 123 for reproductibility\n",
    "# https://pandas.pydata.org/pandas-docs/version/0.25/generated/pandas.DataFrame.sample.html\n",
    "# https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.split.html\n",
    "train_data, test_data, _ = np.split(data.sample(frac=1, random_state=123), \n",
    "                                                  [int(0.95 * len(data)), int(len(data))])  \n",
    "\n",
    "# Save to CSV files\n",
    "train_data.to_csv('automl-train.csv', index=False, header=True, sep=',') # Need to keep column names\n",
    "test_data.to_csv('automl-test.csv', index=False, header=True, sep=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -l automl*.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**No preprocessing needed!** AutoML will take care of this, so let's just copy the training set to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.upload_data(path=\"automl-train.csv\", key_prefix=prefix + \"/input\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the SageMaker AutoPilot job\n",
    "\n",
    "After uploading the dataset to S3, we can invoke SageMaker AutoPilot to find the best ML pipeline to train a model on this dataset. \n",
    "\n",
    "The required inputs for invoking a SageMaker AutoML job are the dataset location in S3, the name of the column of the dataset you want to predict (`y` in this case) and an IAM role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_config = {\n",
    "    'CompletionCriteria': {\n",
    "      'MaxRuntimePerTrainingJobInSeconds': 600,\n",
    "      'MaxCandidates': 10,\n",
    "      'MaxAutoMLJobRuntimeInSeconds': 3600\n",
    "    },\n",
    "}\n",
    "\n",
    "input_data_config = [{\n",
    "      'DataSource': {\n",
    "        'S3DataSource': {\n",
    "          'S3DataType': 'S3Prefix',\n",
    "          'S3Uri': 's3://{}/{}/input'.format(bucket,prefix)\n",
    "        }\n",
    "      },\n",
    "      'TargetAttributeName': 'y'\n",
    "    }\n",
    "]\n",
    "\n",
    "output_data_config = {\n",
    "    'S3OutputPath': 's3://{}/{}/output'.format(bucket,prefix)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launching the SageMaker AutoPilot job\n",
    "\n",
    "We can now launch the job by calling the `create_auto_ml_job` API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import gmtime, strftime, sleep\n",
    "timestamp_suffix = strftime('%d-%H-%M-%S', gmtime())\n",
    "\n",
    "auto_ml_job_name = 'automl-dm-' + timestamp_suffix\n",
    "print('AutoMLJobName: ' + auto_ml_job_name)\n",
    "\n",
    "sm.create_auto_ml_job(AutoMLJobName=auto_ml_job_name,\n",
    "                      InputDataConfig=input_data_config,\n",
    "                      OutputDataConfig=output_data_config,\n",
    "                      AutoMLJobConfig=job_config,\n",
    "                      RoleArn=role)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tracking the progress of the AutoPilot job\n",
    "SageMaker AutoPilot job consists of four high-level steps : \n",
    "* Data Preprocessing, where the dataset is split into train and validation sets.\n",
    "* Recommending Pipelines, where the dataset is analyzed and SageMaker AutoPilot comes up with a list of ML pipelines that should be tried out on the dataset.\n",
    "* Automatic Feature Engineering, where SageMaker AutoPilot performs feature transformation on individual features of the dataset as well as at an aggregate level.\n",
    "* ML pipeline selection and hyperparameter tuning, where the top performing pipeline is selected along with the optimal hyperparameters for the training algorithm (the last stage of the pipeline). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "job_status = job['AutoMLJobStatus']\n",
    "job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "\n",
    "if job_status not in ('Stopped', 'Failed'):\n",
    "    while job_status in ('InProgress') and job_sec_status in ('AnalyzingData'):\n",
    "        job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "        job_status = job['AutoMLJobStatus']\n",
    "        job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "        print (job_status, job_sec_status)\n",
    "        sleep(30)\n",
    "    print(\"Data analysis complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing notebooks generated by SageMaker AutoPilot\n",
    "Once data analysis is complete, SageMaker AutoPilot generates two notebooks: \n",
    "* Data exploration,\n",
    "* Candidate definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "job_candidate_notebook = job['AutoMLJobArtifacts']['CandidateDefinitionNotebookLocation']\n",
    "job_data_notebook = job['AutoMLJobArtifacts']['DataExplorationNotebookLocation']\n",
    "\n",
    "print(job_candidate_notebook)\n",
    "print(job_data_notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's copy these two notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh -s $job_candidate_notebook $job_data_notebook\n",
    "aws s3 cp $1 .\n",
    "aws s3 cp $2 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go back to the folder view, and open these notebooks. Lots of useful information in there!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing all candidates explored by SageMaker AutoPilot\n",
    "Once model tuning is complete, you can view all the candidates (pipeline evaluations with different hyperparameter combinations) that were explored by AutoML and sort them by their final performance metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "job_status = job['AutoMLJobStatus']\n",
    "job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "\n",
    "if job_status not in ('Stopped', 'Failed'):\n",
    "    while job_status in ('InProgress') and job_sec_status in ('ModelTuning'):\n",
    "        job = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "        job_status = job['AutoMLJobStatus']\n",
    "        job_sec_status = job['AutoMLJobSecondaryStatus']\n",
    "        print (job_status, job_sec_status)\n",
    "        sleep(30)\n",
    "    print(\"Model tuning complete\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "candidates = sm.list_candidates_for_auto_ml_job(AutoMLJobName=auto_ml_job_name, \n",
    "                                                SortBy='FinalObjectiveMetricValue')['Candidates']\n",
    "index = 1\n",
    "for candidate in candidates:\n",
    "  print (str(index) + \"  \" \n",
    "         + candidate['CandidateName'] + \"  \" \n",
    "         + str(candidate['FinalAutoMLJobObjectiveMetric']['Value']))\n",
    "  index += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect SageMaker AutoPilot trials with Amazon SageMaker Experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SageMaker AutoPilot automatically creates a new experiment, and pushes information for each trial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.analytics import ExperimentAnalytics, TrainingJobAnalytics\n",
    "\n",
    "exp = ExperimentAnalytics(\n",
    "    sagemaker_session=sess, \n",
    "    experiment_name=auto_ml_job_name + '-aws-auto-ml-job',\n",
    ")\n",
    "\n",
    "df = exp.dataframe()\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploying the best candidate\n",
    "Now that we have successfully completed the AutoML job on our dataset and visualized the trials, we can create a model from any of the trials with a single API call and then deploy that model for online or batch prediction using [Inference Pipelines](https://docs.aws.amazon.com/sagemaker/latest/dg/inference-pipelines.html). For this notebook, we deploy only the best performing trial for inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best candidate is the one we're really interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_candidate = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)['BestCandidate']\n",
    "best_candidate_identifier = best_candidate['CandidateName']\n",
    "\n",
    "print(\"Candidate name: \" + best_candidate_identifier)\n",
    "print(\"Metric name: \" + best_candidate['FinalAutoMLJobObjectiveMetric']['MetricName'])\n",
    "print(\"Metric value: \" + str(best_candidate['FinalAutoMLJobObjectiveMetric']['Value']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the containers and models composing the Inference Pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for container in best_candidate['InferenceContainers']:\n",
    "    print(container['Image'])\n",
    "    print(container['ModelDataUrl'])\n",
    "    print('-')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = 'automl-dm-model-' + timestamp_suffix\n",
    "\n",
    "model_arn = sm.create_model(Containers=best_candidate['InferenceContainers'],\n",
    "                            ModelName=model_name,\n",
    "                            ExecutionRoleArn=role)\n",
    "\n",
    "print('Best candidate model ARN: ', model_arn['ModelArn'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's deploy the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# EndpointConfig name\n",
    "timestamp_suffix = strftime('%d-%H-%M-%S', gmtime())\n",
    "epc_name = 'automl-dm-epc-' + timestamp_suffix\n",
    "\n",
    "# Endpoint name\n",
    "ep_name = 'automl-dm-ep-' + timestamp_suffix\n",
    "variant_name = 'automl-dm-variant-' + timestamp_suffix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ep_config = sm.create_endpoint_config(EndpointConfigName = epc_name,\n",
    "                                      ProductionVariants=[{'InstanceType':'ml.m4.xlarge',\n",
    "                                                           'InitialInstanceCount':1,\n",
    "                                                           'ModelName':model_name,\n",
    "                                                           'VariantName':variant_name}])\n",
    "\n",
    "\n",
    "create_endpoint_response = sm.create_endpoint(EndpointName=ep_name,\n",
    "                                              EndpointConfigName=epc_name)\n",
    "print(create_endpoint_response['EndpointArn'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "sm.get_waiter('endpoint_in_service').wait(EndpointName=ep_name)\n",
    "\n",
    "resp = sm.describe_endpoint(EndpointName=ep_name)\n",
    "status = resp['EndpointStatus']\n",
    "\n",
    "print(\"Arn: \" + resp['EndpointArn'])\n",
    "print(\"Status: \" + status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring the best candidate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's predict and score the validation set. We'll compute metrics ourselves just for fun."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tp = tn = fp = fn = count = 0\n",
    "\n",
    "with open('automl-test.csv') as f:\n",
    "    lines = f.readlines()\n",
    "    for l in lines[1:]:   # Skip header\n",
    "        l = l.split(',')  # Split CSV line into features\n",
    "        label = l[-1]     # Store 'yes'/'no' label\n",
    "        l = l[:-1]        # Remove label\n",
    "        l = ','.join(l)   # Rebuild CSV line without label\n",
    "                \n",
    "        response = sm_rt.invoke_endpoint(EndpointName=ep_name, ContentType='text/csv', Accept='text/csv', Body=l)\n",
    "\n",
    "        response = response['Body'].read().decode(\"utf-8\")\n",
    "        #print (\"label %s response %s\" %(label,response))\n",
    "\n",
    "        if 'yes' in label:\n",
    "            # Sample is positive\n",
    "            if 'yes' in response:\n",
    "                # True positive\n",
    "                tp=tp+1\n",
    "            else:\n",
    "                # False negative\n",
    "                fn=fn+1\n",
    "        else:\n",
    "            # Sample is negative\n",
    "            if 'no' in response:\n",
    "                # True negative\n",
    "                tn=tn+1\n",
    "            else:\n",
    "                # False positive\n",
    "                fp=fp+1\n",
    "        count = count+1\n",
    "        if (count % 100 == 0):   \n",
    "            sys.stdout.write(str(count)+' ')\n",
    "            \n",
    "print (\"Done\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Confusion matrix\n",
    "print (\"%d %d\" % (tn, fp))\n",
    "print (\"%d %d\" % (fn, tp))\n",
    "\n",
    "accuracy  = (tp+tn)/(tp+tn+fp+fn)\n",
    "precision = tp/(tp+fp)\n",
    "recall    = tn/(tn+fn)\n",
    "f1        = (2*precision*recall)/(precision+recall)\n",
    "\n",
    "print (\"%.4f %.4f %.4f %.4f\" % (accuracy, precision, recall, f1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does this compare to what you achieved in the first lab?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deleting the endpoint\n",
    "Once that we're done predicting, we can delete the endpoint (and stop paying for it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment to delete\n",
    "sm.delete_endpoint(EndpointName=ep_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SageMaker AutoML job creates many underlying artifacts such as dataset splits, preprocessing scripts, preprocessed data, etc. Let's delete them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_outputs_prefix = '{}/output/{}'.format(prefix,auto_ml_job_name)\n",
    "print(job_outputs_prefix)\n",
    "# Uncomment to delete\n",
    "#bucket.objects.filter(Prefix=job_outputs_prefix).delete()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
